Grid Computers, Computer Farms and similar Computer Clusters are being used to deploy a novel type of parallel applications based on concurrent independent tasks related only by their individual contribution to a global problem's resolution. Until now all parallel applications were based on splitting a single task into a multitude of collaborating subtasks (i.e. OpenMP, PVM, MPI, etc). However, in some application areas users have recently started to split single large problems into a multitude of sub problems which can be resolved independently of one another. This methodology allows higher scalability and permits the use of Grid Computing techniques and the use of cost efficient computing solutions (i.e. clusters), but requires that the necessary data files first be replicated to the remote nodes prior to the computation taking place. It is this problem of replicated data transfers that our invention addresses.
Existing art to address data file transfer falls into three categories.
First, tasks can make use of on-demand file transfer apparatus, better known as file servers. For problems where file access is minimal, this type of solution works as long as the cluster size (i.e. number of remote computers) is limited to a few hundred. For large and frequent file accesses, this solution does not scale beyond a handful of nodes. Moreover, if entire data files are accessed by all nodes, the total amount of data transfer will be N times that of a single file transfer (where N is the number of nodes). This waste of network bandwidth limits scalability and penalizes computational performance as the nodes are blocked waiting for remote data.
Second, users or tasks can manually transfer files prior to execution through a point-to-point file transfer protocol. There are three types of point-to-point protocols. Standard file transfer protocols (i.e. ftp, tftp) where one file is transferred to one remote node, one packet at a time. Sliding window file transfer protocols, such as the “parallel file transfer protocol” from Donald J. Fabozzi II where multiple packets transit concurrently on their way to a single remote node. And parallel file transfer protocols (ex HPSS PFTP) where multiple point-to-point file transfers operate concurrently. While these methods improve network bandwidth utilization over demand based schemes, the final result is the same: a file is transferred “N” times over the network when replicating information unto “N” remote computers. Moreover, additional file transfers must continually be initiated to cope with the constantly varying nature of large computer networks (i.e. new nodes being added to increase a cluster or Grid size or to replace failed or obsolete nodes).
Third, users or tasks can manually transfer file prior to execution through a multicast (or broadcast) file transfer protocol (ex StarBurst SMFTP). In this scheme each file fragment sent over the network is simultaneously read by all participating remote computers. Hence network bandwidth usage is limited to the same amount of data traffic as for a single point-to-point file transfer. This is currently the most frequent scheme used to resolve problems having been split into multiple concurrent independent tasks as described above. However, this form of apparatus is imperfect. For instance, error recovery is concurrent to the multicast phase. This imposes an increased workload on the master file server node and eventually will limit scalablity. These schemes also are based on the notion of node registration, where prior to the multicast phase, all active and participating remote computers must register to participate in a transfer request. Hence, new nodes being booted during or after the multicast transfer phase will not be participating in the effort to replicate files. Another drawback is that registered computers which crash during the multicast phase cannot join back the transfer group after reboot. Finally, these schemes cannot survive through a crash on the master file server (i.e. the computer which performs the multicast file transfer). These sum of these limitations is that current multicast file transfer art work fail at their task of insuring correct file replication among all participating remote computers in a normal setup of dynamic and error prone network of computers. They lack the fault-tolerance, ability to handle dynamic registration, scalability to tens of thousands of nodes and capability to persist with the file replication effort once the master transfer process terminates.